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of research that objectively states research questions, hypotheses, 
methodology, and results. (Author/EL) 
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WHAT SOME JOURNAL REFEREES tOOK 



FOR IN EVALUATING MANUSCRIPTS 



It's a struggle for some writers to prepare a m^uscrlpt for submission 
. to a scholarly Journal in Joumal5^.slp and cy^mmunication. Particularly rflnce 
there isn't much concrete guidance available to those, who want to write. 

This * paper' represents an effort establish some gtildellnes <f or 
scholars who want to publish* y 

The guidelines were developed through a survey of 29 randomly selected 
nembere of the editorial boards of the Journal of Broadcasting , the Journal 
of Cotmntinication and Journalism Quarterly . Sixteen persons responded, for a 
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return rate of 55*2X. \ / v 

\ • C' \ ' • 

The sample Include^ /seven of the Journal pf Broadcasting's editorial 
consultants, eight of Journalism Quarterly's editorial advisory board members 
and 14 of the Journal of Communication' s consulting anA. contributing editors. 
Four of the Journal of Broadcasting's editorial consultants, five of Journalism 
Quarterly's editorial advisory l^ard members and seven of the Journal of Commu- 
nication's consulting and cpntributing editors responded.^ , V ^ 

• ' 1 • ■ * i" , • ■ - ' 

/ Respondents were asked to pftate in their own words "...the specific things 



you look for as .you evaluate manuscripts inu 
advisor." They also were asked: 

... » to ^mention not ^nly the igajor 



ybur role as an editorial" consultant/ 



strength of the research design, th(5 completeness of the overall. report, 
etc.) but also the small things that, when added up^ can'make a 'difference 
between acceptance and rejection (f>r example, writing ^fejrle, typos. 



etc.). . 



Results dearly indicate that 'different reviewers use somewhat different 



criteria as th^ evaluate manuscripts. And 



things you look fot (such aaf the 



• ■ - ■ « 

it may be that acceptance or rejection 



of a manuscript depends in part on "the luck of the draw." Pet, one reviewer 
and the manuscript is accepted; get another and it ,i8 rejected. . 

^ ^ [ ♦ , ^ ■ ^ 

Introductory Material ^ 

At the same, time, howler, there is some consistency among reviJbwers. 

For otle thing, most respot|denta seemed to agree with Phillip J. Tlcjienor, 

^ number of Journalism Quarterly ' s editorial advisory board, who' said! 

The fKst^lement that most reviewers look for, in my experience. 



/ is a staifement of &:\(^ problem and of the perspective,- or theoretical 

. ' ' . 

^ framework, which is to be employed in studying that problem. /v' . 

Statements of hypothed^s should be clear and^uppprted by the theo- 

, retical persrpective being used and, if appropriate, by previous 

• • r ■ ' # . ; ■ • ■ . 

felevant\work. 

And mos^ agreed also that the questions addressed should be of some 

s^^ificance. A coiAment by Nathan Maccoby^^ one of the Journal of Commtmi- 

' . , - ■ • ■ * 

cation's consulting and contributing editors^ Is typical: , 

I'suppose, first of all, I am looking for something which is a , 

contribution to knowledge, *feven if it is a little one. Even if it ' ' 

is a replicajtion of something that has been previojzsly done, if it 

is well done and if it is something that obviously naeds replication, 

that I would regard as. an addition' to knowledge. I hope it Jias some 

^ th'eprltical interest and importeact because that would make it sonie- ^ 

\ how more generally' Interesting arid applicable and useful.' ^ ' / 

It is imjfortant* for au^jtiors to avoid trivial questions in their researcly-os^ ^ 

But authors must do more than sin?>ly pick worthwhile topics; they must know^ , 

... ) • ' ■ 

how 1:0 convince others* that the top*c' Is worthwhile.. This can be done . - 

thtbugh a solid literature re-^iew. As Ivan Preston, a member of Journalism 
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^Qii arteifly^s editorial advisory board, said: 7 ' 



(3) . 

One more thing I'd a4d as advice to authors: always go quickly, 

V 

after a proper introduction, into an explanation of what your article 
contributes — what it covers that hasi ^ not been covered before. Quickly 



summarize the existing literature to show that knowledge has been 
carried so far and no further, and then show how your contribution extends 
knowledge further. in such-and-such precise way. This is where readers 
assess the worth of research, and authors should assist readers in 
• doing so, and in fact do it for them. 
A good literature 'review will not only help a scholar Justify the^ research 
^^V^ question he or she is studying, it also will help put his or her sttMy in 
context. . It will help the writer meet a criterion for historical* studies 



s 



specified by Christopher H. Sterling, one of the Journal of Broadcasting' 
editorial consultants, who said: "I look for some connection to a current 
audi ence--why should we care about this event, or. trend, or person today? 



>^ What does this event tell us about present day policy or events?". 
■^^^ A well-done literature review must be base^d on the best available source 



-.y{ p inatcrial,' according to Richard A. Schwarzlose, a meinb'er of Journalism Quarterly's 



) - . ■ 

~; editorial- advisory board: * 



Students are often confined to llbrarspcollectlons and the 
■Inevitable reliance on secondary Rtnirces . But Journalism Quarterly * 

. ■ ■ . ' : ^- ■ V ■ ■ ' \ - 

is the big time in our discipline* and. the contributors have a special . 
obligation to present the most detailed, accurate, and penetrating 
source material available. T read a mkhuscripftv with th6^<*notes 
at my elbow; I want to know at every Sytep where the material, comes 

■ " from. ■ ■ ''^ , 

■ ■ ■ . - \ ' \ - 

A thorough literature yreview also will help an author Insure that he or . 

Bt^ doesnVt (^^esearch that already has been published elsewhere. / 
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For Judith Beinstein, A consulting editor for the Journal of Communijcation. 
accuracy is as important as thoroughness. "I also note [an author 'sjintlerpre- 
fiitlon or use of previous literature, " she said. "If. for example, much of his 
argument depends on several past studies, I want to be sure^hifl- explana^ion^of 

those studies is accurate." * i ^ 

» V ' . . . ■ \ 

Methodology ' > 

• A scholar who has a question worth asking must select an appropriate 
methodology to answer the question. Most respondents seemed to agree with 
Vivian M. Homer. an editorial consultant for the Journal of BroadcastinR. 
who "...would like to see a research design^which allows the maximum rigor 
permitted by the question (and I confess jo a suspicion that there is an 
inverse relationship betwee^i the importance of the question asked and the 



) 



r 



degree of rigor possible) . " > 

But what exactly should .ai^Jluthor say about a research design? Soma 

reviewers loo^c for .a.nswers to some specific questions as they read articles 
8ubmitte;d for^poasible publication. Questions asked by Roger Wimmer, an 
editorial consultant for the Journal of Broadcasting , include: 

Is the research method used correctly, and is it accurately 
' ' . \ "kescrJied?- (This is a major problem in many research studies 

in^ded for publication.) Would another statistical procedure 
be better stiited for the research question? Has the author 

violated any assumptions or . procedures of. a statis'tical method? 

% ■ ' " ■ ■ 

Is the sample explained in -detail? Could another author. • 

duplicate the sample? Is the random selection explained* well? 

An4 Hugh M. Culbertspn, a member of Journalism Quarterly's, editorial 

advisory board, seeks answers to these questions: ^ 

Is it clear Just how variables are measur^ed? I like to s^e at 

' . ■ ' ■ * ■■ ' ' , ■ 

least a couple of sample items from a large index, with clear 
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■ , ^/ • 

explication of how scoring ±b done. 1 find this to be a coramon 

problem area**** , ^ 

^ Do the operational definitions measure what they are presented aa 

measuring? Predictive and/or construct validity is seldqpi established 
for independent variables in our field. Also, mowe kind of convergent 
validity with con^anion measures is seldom established for dependent 
variables. In light of these points,' assessment of . operational 
definitions tends to boil down to an intuitive or common-sense look 
at "face" validity. (I would suggest that statistical-inference 
procedures tend to be overextphasized, ipeasirement proceduifes uti^r- 
emphasized, in mass communication research. The first part of this 
sentence mi^t be challenged. I don't .think the second part could 
be.) . ' . . ' 

Refl<utts ' 

The main thing these reviewers seem to locjk for in an adequate results 

sectdlon is accuracy'. "I check for accuracy of tables, figuresi, data," said 

♦ \ «^ 

✓ Rita Wicks Poulos, one of the Journal of Broadcasting's editorial consultants. 
"They must be accurate. Obviously, an inconsistency, in data would require at 
least ^^clarification and. correction — and reevaluation of the paper." 

Authors definitely should check the results section for internal 
accuracy. Most reviewers surveyed here said they always com>a^ the numbers 

^ reported in the tables with those j/eported iii t^e text and they always total 
numbers in tables (where approptlate) ta make iertain the results are what 
they should be. Substantial' Inaccuracies in the results section can reisult^ 

In rejection of an otherwise acceptable manuscript: 

. '^^ ■ . ■ •. * 
Authors should be care'^ul to Insure their tables are no b^nly accurate, 

•N^ . ■ >^ ■ ■ , ■ " ■ ■ ^ . ; 

but also necessary. Tables that do not contain importani; or relevant material 
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should be eliminated; and authors^should insure that tables are presented in 
the best; most logical and most efficient manner • As Roger Wimroer said: 

^ar»-do not en^oy-complex tahlew nnd graphs because_they_£Q8t a_lot_jto : 

produce. Authors should decide whether a table is absolutely necessary; 
if not, they can include a footnote indicating where the information is 
available." ' * ' * ) 

Hugh Culbertson maintains that results generally should be repo:i^ted 
in such a manner that one can': , , — 

(a) Identify what data are pertinent to a priori hypotheses. 

(b) Identify what conclusions are supported unequivocally, and 

what conclusions- tequire some qualification. If qualification is 

needed, it should be made explicit. 

"•(c) Assess the generalizability of findings, I do not believe 
• • • 

a study has to have extremely high generalizability (for example, 

•* applicability to the entire nation or world) to be o| value. But I do 

believe that, where generalizability .•. is limited, the author shpuld dis- 
cuss ihe issue. I think it appropriate in survey^ research, by the way, 
to present some jJemographic comparisons between sample data and Census- . ' 
based population data to establish the adequacy of sampling. 
Roger Wimmer mentioned one other matter regarding the presentation of - • 
results that authors might consider as ^they prepare articles for submission: 

Is th^re a^' footnote indicating where RAW data and other- releyant in- 
fqrmation iay be obtained?^ THIS IS" VERY. IMPORTANT. Any published re- 
search stud|^ should contain enough information to^ allow* an independent 
check of results by another researcher. To date, this has not been folldw- 
^' ed by many editors, but gaining in acceptance. This me^ns ;that authors 
should include addresses, data archives; or any place where people can 
. flret^ their hands on the data. , ^ . • 



' This «i«o has baen a problam^ I %ibuld guaaa that 99 out of 100 
authors would not have theit raw, data available for othet researchers 
to use» This can oply mean one of two things: The research was never 



complete'd, or the researcher Involved has not allowed strict guidelines 
and Is paranoid about others seeing what he or she did.... 

Conclusions ' * , ^ * 

Reviewers — at least the ^ ones surveyed 'here — seem to put a high preiAuni on 
honesty and accuracy In research reports. And honesty and accuracy are as 
Important In' conclusions sections as In other parts of^a research ^report . 
As Vivian Horner said: 

Sometimes even good research designs don't allcyw you to .answer the 
questions you asked. I like* to see, an honest admission of that w^en 
it happens and a straightforward statement of what ma7~have turned up 
anyway. Serendipitous findings are, of ten the most lln teres ting. And I 
find fewer things more tedious than , wading through^statistical pyrotechnics 
performed on Irrelevant data. 
Roger Wimmer jseconded Ms. -Homer's comments, saying: 

It seemsythat many authors try to make^ their stud^be somethfng that 
it really isn't. Some authors tty to make their article, which may be 
good, to be something earth shattering. They don't stick with the data 
they have and often times miss an important small point in an effort to 
^ discuss a major point which really doesn't exist. 
4wo recommendations were made by Hu^h Culbertson for improving the 
conclusions sections written for scholarly Journals in communications. For one 
thing, he said, an author , should try to show how results of his or her study 
dovctailTwlt^ or relate to results reported in other studies. "'Does it fill ^ 
an existing gap or represent an extension of prior work?" he asks as he evaluates 



msinuscrlpts. 
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Furthennore, he asks: "Are practical wplicatlona made clear, even if 
they seeb tentative? I think this Is more Important for Joutnaltsm Quarterly. 
In light of Its broad and basi cally 'applied' re adership, than for some other 

Journals.'* , 
Finally, Richard Schwarzlose cited as a weakness In some concluslona 
' sections "...the problem too many authors have of overgenerallzlng their 
data, making assumptions about people's motives,^ expressing or Implying a 
bias which Is a more personal than research approach." 

Writing 

A major problem with some manuscripts Is that the authors haven't taken 
time to read the Journal carefully to determine whether that journal Is the 
article's' best market and to learn the journal's style. 

Failure tb tailor a mafiusorlpt to the journal suggests, according iCo 
Hugh Culbertsqn, "...that (a) a person hasn' t read the journal carefully. 



^ (b) he or she ^as sent? the manuscript elsewhere and Is sending to the current 
journal as a second or third ftholce— clearly not a wise ^hlng to 'telegraph," 



or (c> both ^ and b." There Is no evidence In this study that failure to 
tailor an article to the jounrnl will necessarily result In rejection, but a 
marginal piece that Isn't tailored to the journal may not stand a chance. 

Almost all of the editorial board members surveyed said they were dis- 
turbed about'^the^ltlng problems evident In mmy manuscripts, but they 
apparently would not often rteject a* manuscript for poor writing alone. "I 
have not found that writing /style leu a significant contributor to my 
recommendatl6n," Charles WlMck,^|j||^suIttng e41tor for the Journal of 
Communication , said, for In^ance^But he added: "Very few articles have typos, 
although I suspeit that I would be negatively Influenced py any such si^n of ' 
carelessness on the part of. an author. ' \^ 
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Ivftn Pi^aton mad* • similar point whan ha a§ld: 



Of couraa [artlda praaahtatlon] problama can ba aolvad by 
favrlta, so I naver racoimand rajactlon on auch g^rounda alona^ 



^ But I alao baliava that an obacura atyle may kaap ma from aaaeaalng 
tha quality of a -place 'a conteiit, ao that I am mora likely to reject 
the piece on ^ther grdunds. 
The message? lake the time to submit a well-wrj.tten manuscript. That 
means avoiding what Richard Scljwartlose called a general lack of thbughtful- 



ness: 



A piece must say what it will do, how it will do it, do it, and 
then say something about ^what was (or wasn't) done. All top often a 
' piece promises more than it delivers (or perhaps can deliver). 

Pieces sometimes mysteriously change subject or focus in midstream. 

' • ^ ■ 

Such problems reflect a lack of proper focussing and conceptiializing 

• c 
before and during the writing. ^ 

A strong introduction can help a writer keep an article carefully focussed 

and can help a reader follow a writer through even complex material. Bufj some 

respondents said strong introductions are rather rare. Ivan Preston's* comment 

is typical: 

I can distinctly observe that the first thing I look for in many 
manuscripts — or more accurately, the first thing I see— because it 
jumps off the page at me before there's any conscious effort to look — 
Is that a writer often does a terrible job of introdi^ng his article. 
It's incredible how many people start out with background comments, 
which usually play a useful role, and then get around to telling you 
what their study will encompass only on page three or four. In about 
two-thirds of 'the papers I review I generally lo9ate some sentence on 
* one of those later pages which ought to be the very first sentence in the 
" * article. ... 
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. Finally » • writer noads to avoid flimsy footnoting) oKCMsive or 

• 41^ ' ■ 

unaacMMry diract quotation; apalling, granawitical and typographical orrora; 

and torturoua and varboaa writing, "Ona'can ba accurate and datailcd in 

technical writing without baing terribly wordy and unreadable," Hugh Culberteon 
aaid. "Unfortunately, this takea work which all too many authora seem unwilling 
tp put forth." 

Advocacy Reeearch 

• Some revicwcre see the relatively recent rise of "advocacy reeearch" as a ' 
major problem with a few manuecriptg. Reeearchera occasionally are quite close 
to the problems or issues they are st^jjdying and they allow their own opinions 
to enter their articles. 

The authors of an article aUout media cpverage of the U.^. Supreme Court's 
deciaion in Rog vs. Wade , for instance, described the court's decision as a 
critical event. And they said: / • • 

It C-he decision] followed years, even decades of .debate on a 
aignif leant topic of surpassing moral, legal Aid political contro- 
versyand it dealt directly with one of the most central questions 
affecting the movement for female equality, namely: to what extent ^ 
do women have the right to control their own bodies? , ^ 
The authors failed to point out the question also is central to the pro- 
life movement, and that the question to those who oppose abortion might be 
stated as: how long will genocide continue to be practiced against unborn 

/ 

perscins? • * / ^ 

The point is iwt that the authors were bia^ped in their research (they may 
or may not have been) or that they should have e3q>lained the al«Jr.tion controversy 
in detail. The point is that the one-sided vay in w.hich the question was stated 
Indicated a possibility of bias on the part of the researchers. 

'A ^ . 

\ • • 1 o 
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, An indication of bi«s irT a manuscript apparently causey some reviewers 
to be ski^tical of the article. If bias creet>s into a xeport of a ^tudy, a \ 
reviewer Mght sujpect^ t^ research methods. And few * 

journal referees are* likeiy to'reconmend publlcatio of an article that indicates 

. . . •■' -S* f • • 

the use of biai^jed- nieithods. - • 

Scholi^iwboi^want ' their articles published probably should follow the 
advice jour^lism .teachers often give their students: be as objective and as 
fair as you can in collecting" and reporting informatipA. Try as hard as you can 

' 0 , 

to control^your own biases. If you can't control your biases, don't write the 
story. 
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^^^^ The- 29 persons were selected frpm modified lists of ^editorial board 
members for each journal* / Persons associated with colleges^ universities or 
Other agencies outside the United States were excluded because of the ^ . 

difficult ies'^ISiblyed in getting materials to and responsea f rom euch persons. 
One men&er of the Journal of Broadcasting^s board of editorial consultants an<^--^ 

56 of the Journal of Communication's consulting and cqntribu ting editors were 

■ • ■ * '\ 
eliminated from the population* » • . 

^ • • ' ' . ." .■ ■ 

\ 2 ' ' ' ■ ■ • - ' ' 

Appro3d.mately 25% of each journal 's. editorial bo^rd members (excluding 

those affiliated with institutions outside the United States) were contacted 

for this research'. ; ^ 

" 3 .■ • ' •' ' . • . S"' ■ 

This problem is mitigated to some extent by the fact that most journal 

' . ■ ■ ■ ■ ■ . . • 

editoVs send manuscripts to at least two different reviewers. - 

r V • ■: ■ . ' ■ ' ■ : ■ , ■ ■ : ' 

This, of cojurse, doesn't mean that completed questionnaires should be made 
available to other researchers. As Roger Wimmer pointed out, one wpuld need, the 
permission of each person who filled out a questionnaire before questionnaires 
could be made available to other researchers. However, summary material and 
pmiched cards (without names) could be made available, - -i; . - \ ' 

^John Crothers Pollock, James Lee Robinson, Jr. and Mary Carmel \ 
"Me*dia Agendas and Human Rights: The. Supreme Court Decision on Abortion," 
Journalism Quarterly . 55:544-48, 561 (Autumn 1978) , p. 544. 
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